Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HDDS-2294. Create a new HISTORY.md in the new repository. #764

Merged
merged 7 commits into from
Apr 10, 2020
Merged
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
60 changes: 60 additions & 0 deletions HISTORY.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
<!---
elek marked this conversation as resolved.
Show resolved Hide resolved
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->

# History of Apache Hadoop Ozone project

Ozone development is started on a feature branch of HDFS-7240 as part of the Apache Hadoop HDFS project. Based on the Jira information the first Ozone commit was the commit of [HDFS-8456 Ozone: Introduce STORAGE_CONTAINER_SERVICE as a new NodeType.](https://issues.apache.org/jira/browse/HDFS-8456) in the May of 2015.
elek marked this conversation as resolved.
Show resolved Hide resolved

Ozone is an Object Store for Hadoop which is based on a lower level storage replication layer. This layer is called originally HDSL (Hadoop Distributed Storage Layer) and renamed later to HDDS (Hadoop Distributed Data Storage).
elek marked this conversation as resolved.
Show resolved Hide resolved

This generic storage layer started be implemented under [HDFS-11118](https://issues.apache.org/jira/browse/HDFS-11118) together with a jScsi based block storage layer ("CBlock") introduced by [HDFS-11361](https://issues.apache.org/jira/browse/HDFS-11361).
dineshchitlangia marked this conversation as resolved.
Show resolved Hide resolved

As a summary:

* HDDS (earlier HDSL): replicate huge binary _containers_ between datanodes
elek marked this conversation as resolved.
Show resolved Hide resolved
* Ozone: provides Object Store semantics with the help of HDDS
* CBlock: provides mountable volumes with the help of the HDDS layer (based on iScsi protocol)

In the beginning of the year 2017 a new poddling project is started inside [Apache Incubator](http://incubator.apache.org/): [Apache Ratis](https://ratis.apache.org/). Ratis is an embeddable RAFT protcol implementation it is which become the main corner stone of the consensus inside Ozone/HDDS projects both (Started to [be used](https://issues.apache.org/jira/browse/HDFS-11519) by Ozone at March of 2017)
elek marked this conversation as resolved.
Show resolved Hide resolved

In the October of 2017 a [discussion](https://lists.apache.org/thread.html/3b5b65ce428f88299e6cb4c5d745ec65917490be9e417d361cc08d7e@%3Chdfs-dev.hadoop.apache.org%3E) has been started on hdfs-dev mailing list to merge the existing functionality to the Apache Hadoop trunk. After a long debate Owen O'Malley [suggested a consensus](https://lists.apache.org/thread.html/c85e5263dcc0ca1d13cbbe3bcfb53236784a39111b8c353f60582eb4@%3Chdfs-dev.hadoop.apache.org%3E) to merge it to the trunk but use separated release cycle:
elek marked this conversation as resolved.
Show resolved Hide resolved

> * HDSL become a subproject of Hadoop.
> * HDSL will release separately from Hadoop. Hadoop releases will notcontain HDSL and vice versa.
elek marked this conversation as resolved.
Show resolved Hide resolved
> * HDSL will get its own jira instance so that the release tags stay separate.
> * On trunk (as opposed to release branches) HDSL will be a separate module in Hadoop's source tree. This will enable the HDSL to work on their trunk and the Hadoop trunk without making releases for every change.
> * Hadoop's trunk will only build HDSL if a non-default profile is enabled. When Hadoop creates a release branch, the RM will delete the HDSL module from the branch.
dineshchitlangia marked this conversation as resolved.
Show resolved Hide resolved
> * HDSL will have their own Yetus checks and won't cause failures in the Hadoop patch check.
elek marked this conversation as resolved.
Show resolved Hide resolved

This proposal is passed and after reorganizing the code (see HDFS-13258) and Ozone [has been voted](https://lists.apache.org/thread.html/ad0fe160ae84be97a0a87865059761ad7cd747be7b2fe060707d4f28@%3Chdfs-dev.hadoop.apache.org%3E) to be merged to the Hadoop trunk at the March of 2018.
elek marked this conversation as resolved.
Show resolved Hide resolved

As the CBlock feature was not stable enough it was not merged and archived on a separated feature branch which was not synced with the newer Ozone/HDDS features. (Somewhat similar functionality is provided later with S3 Fuse file system and an S3 compatible REST gateway.)
elek marked this conversation as resolved.
Show resolved Hide resolved

After the merge a new Jira project was created (HDDS) and the work was tracked under that project instead of in the child issues of HDFS-7240 issue.
elek marked this conversation as resolved.
Show resolved Hide resolved

In the next year multiple Ozone releases has been published in separated release package. The Ozone source release was developed on the Hadoop trunk, but the Ozone sources are removed from the main Hadoop releases.
elek marked this conversation as resolved.
Show resolved Hide resolved

Originally Ozone depended on the in-tree (SNAPSHOT) Hadoop artifacts. It was required to compile the core hadoop-hdfs/hadoop-common artifacts before compiling the Ozone subprojects. During the development this dependency is reduced more and more. With the 0.4.1 release this dependency has bee totally removed and it became possible to compile Ozone with the help of the released Hadoop artifacts which made it possible to separate the development of Ozone from the main Hadoop trunk branch.
elek marked this conversation as resolved.
Show resolved Hide resolved

At the October of 2019 the Ozone sources are moved out to the [apache/hadoop-ozone](https://github.com/apache/hadoop-ozone) git reposity. During this move the git history has been transformed to remove old YARN/HDFS/MAPREDUCE tasks.
elek marked this conversation as resolved.
Show resolved Hide resolved

* The first git commit of the new repository is the commit which created the new maven subprojects for Ozone (before the trunk merge)
* Some of the oldest Ozone commits are available only from the Hadoop repository.
* Some newer HDDS commits have different commit hash in `hadoop` and `hadoop-ozone` repository.


At the March of 2020 Ozone 0.5.0 is released, the first release which marked with the _beta_tag (earlier releases were alpha).
elek marked this conversation as resolved.
Show resolved Hide resolved