Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Change worskpace project files storage path #7844

Merged
merged 8 commits into from
Dec 22, 2017
Merged
Original file line number Diff line number Diff line change
Expand Up @@ -292,6 +292,11 @@ che.infra.docker.bootstrapper.installer_timeout_sec=180
# Once servers for one installer available - checks stopped.
che.infra.docker.bootstrapper.server_check_period_sec=3

# Enable to perform migration of workpace projects at Che startup.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should it be only for che.infra.docker ? (prefix)

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unfortunately, I was not able to protect such a naming when Tyler did choose another naming technique. I don't remember that anyone was supporting me at that point. Does it seem better now?

# Projects, which are stored in directories named after their workspace name,
# will be stored by workspace id instead.
che.workspace.migrate_workspace_projects_on_startup=true

### INTERNAL
# Remove locations where internal message bus events should be propagated to.
# For debugging - set to retrieve internal events from external clients.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@
public final class Pages {

/** An experimental value used as default page size where necessary. */
private static final int DEFAULT_PAGE_SIZE = 50;
public static final int DEFAULT_PAGE_SIZE = 50;

/**
* Defines an interface for page supplier.
Expand Down
3 changes: 1 addition & 2 deletions dockerfiles/che/entrypoint.sh
Original file line number Diff line number Diff line change
Expand Up @@ -230,8 +230,6 @@ init() {
sudo chown -R ${CHE_USER} ${CHE_LOGS_DIR}
fi


export CHE_WORKSPACE_STORAGE=/data/workspaces
export CHE_DATABASE=/data/storage
export CHE_TEMPLATE_STORAGE=/data/templates
export CHE_WORKSPACE_AGENT_DEV=${CHE_DATA_HOST}/lib/ws-agent.tar.gz
Expand All @@ -246,6 +244,7 @@ init() {
export CHE_DOCKER_IP_EXTERNAL=${HOSTNAME}
fi
### Necessary to allow the container to write projects to the folder
export CHE_WORKSPACE_STORAGE__MASTER__PATH=/data/workspaces
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why do we need to duplicate the property CHE_WORKSPACE_STORAGE ?
so we have
CHE_WORKSPACE_STORAGE__MASTER__PATH=/data/workspaces
and
CHE_WORKSPACE_STORAGE="${CHE_DATA_HOST}/workspaces"

also /data should be replaced by ${CHE_DATA} property

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because these are different paths?

export CHE_WORKSPACE_STORAGE="${CHE_DATA_HOST}/workspaces"
export CHE_WORKSPACE_STORAGE_CREATE_FOLDERS=false

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@
import org.eclipse.che.api.core.model.workspace.Workspace;
import org.eclipse.che.api.core.util.SystemInfo;
import org.eclipse.che.api.workspace.server.spi.WorkspaceDao;
import org.eclipse.che.commons.annotation.Nullable;
import org.eclipse.che.workspace.infrastructure.docker.WindowsHostUtils;

/**
Expand All @@ -37,140 +38,158 @@
@Singleton
public class LocalProjectsFolderPathProvider {

public static final String ALLOW_FOLDERS_CREATION_ENV_VARIABLE =
"CHE_WORKSPACE_STORAGE_CREATE_FOLDERS";
public static final String WORKSPACE_STORAGE_PATH_ENV_VARIABLE = "CHE_WORKSPACE_STORAGE";
static final String ALLOW_FOLDERS_CREATION_PROPERTY = "che.workspace.storage.create.folders";
static final String WORKSPACE_STORAGE_MIGRATION_FLAG_PROPERTY =
"che.workspace.migrate_workspace_projects_on_startup";
static final String WORKSPACE_PROJECTS_MOUNT_POINT_PROPERTY = "che.workspace.storage";
static final String CHE_MASTER_SPECIFIC_WORKSPACE_PROJECTS_MOUNT_POINT_PROPERTY =
"che.workspace.storage_master_path";
static final String SINGLE_MOUNTPOINT_FOR_ALL_WORKSPACES_PROPERTY = "host.projects.root";

private final WorkspaceDao workspaceDao;
private final boolean isWindows;
private final boolean migrationOnStartup;
private final LocalProjectsMigrator localProjectsMigrator;
private final WorkspaceDao workspaceDao;

/**
* Value provide path to directory on host machine where will by all created and mount to the
* created workspaces folder that become root of workspace inside machine. Inside machine it will
* point to the directory described by {@literal che.workspace.projects.storage}.
*
* <p>For example: if you set {@literal che.workspaces.storage} to the /home/user/che/workspaces
* after creating new workspace will be created new folder
* /home/user/che/workspaces/{workspaceName} and it will be mount to the dev-machine to {@literal
* after creating new workspace will be created new folder /home/user/che/workspaces/{workspaceId}
* and it will be mount to the machine with projects volume to {@literal
* che.workspace.projects.storage}
*/
private String workspacesMountPoint;
private final String workspacesMountPoint;

/**
* this value provide path to projects on local host if this value will be set all workspace will
* This value provide path to projects on local host if this value will be set all workspace will
* manage same projects from your host
*/
@Inject(optional = true)
@Named("host.projects.root")
private String hostProjectsFolder;
@Named(SINGLE_MOUNTPOINT_FOR_ALL_WORKSPACES_PROPERTY)
private String singleMountPointForAllWorkspaces;

/**
* If environment variable with name {@link #ALLOW_FOLDERS_CREATION_ENV_VARIABLE} is equal
* (ignoring case) to {@literal false} then this field is set to false. Otherwise it is set to
* true. It is also possible to overwrite with named constant.
*/
@Inject(optional = true)
@Named("che.workspace.storage.create_folders")
private boolean createFolders = true;
@Named(CHE_MASTER_SPECIFIC_WORKSPACE_PROJECTS_MOUNT_POINT_PROPERTY)
private String workspacesMountPointForCheMaster;

@Inject(optional = true)
@Named("che.user.workspaces.storage")
private String oldWorkspacesMountPoint;
@Named(ALLOW_FOLDERS_CREATION_PROPERTY)
private boolean createFolders = true;

@Inject
public LocalProjectsFolderPathProvider(
@Named("che.workspace.storage") String workspacesMountPoint, WorkspaceDao workspaceDao)
@Named(WORKSPACE_PROJECTS_MOUNT_POINT_PROPERTY) String workspacesMountPoint,
@Named(WORKSPACE_STORAGE_MIGRATION_FLAG_PROPERTY) boolean migrationOnStartup,
WorkspaceDao workspaceDao,
LocalProjectsMigrator localProjectsMigrator)
throws IOException {
this.workspacesMountPoint = workspacesMountPoint;
this.workspaceDao = workspaceDao;
this.isWindows = SystemInfo.isWindows();
this(
workspacesMountPoint,
null,
migrationOnStartup,
null,
null,
workspaceDao,
localProjectsMigrator,
SystemInfo.isWindows());
}

@VisibleForTesting
protected LocalProjectsFolderPathProvider(
String workspacesMountPoint,
String oldWorkspacesMountPoint,
String projectsFolder,
boolean createFolders,
@Nullable String workspacesMountPointForCheMaster,
boolean migrationOnStartup,
@Nullable String singleMountPointForAllWorkspaces,
@Nullable Boolean createFolders,
WorkspaceDao workspaceDao,
LocalProjectsMigrator localProjectsMigrator,
boolean isWindows)
throws IOException {
this.workspaceDao = workspaceDao;
this.workspacesMountPoint = workspacesMountPoint;
this.hostProjectsFolder = projectsFolder;
this.createFolders = createFolders;
this.oldWorkspacesMountPoint = oldWorkspacesMountPoint;
this.migrationOnStartup = migrationOnStartup;
this.isWindows = isWindows;
this.localProjectsMigrator = localProjectsMigrator;

if (createFolders != null) {
this.createFolders = createFolders;
}
if (singleMountPointForAllWorkspaces != null) {
this.singleMountPointForAllWorkspaces = singleMountPointForAllWorkspaces;
}
// In case workspace mount path specific for che master is not defined we treat it equal to
// regular workspace mount point
if (workspacesMountPointForCheMaster != null) {
this.workspacesMountPointForCheMaster = workspacesMountPointForCheMaster;
} else {
this.workspacesMountPointForCheMaster = workspacesMountPoint;
}
// Priority of workspace storage path sources:
// If Che is running on Windows
// che-home-location/vfs
// Otherwise
// use value from property injected into constructor
// find root directory for projects in workspaces
if (isWindows) {
final Path vfs = WindowsHostUtils.getCheHome().resolve("vfs");
this.workspacesMountPoint = vfs.toString();
} else {
this.workspacesMountPoint = workspacesMountPoint;
}
}

public String getPath(String workspaceId) throws IOException {
if (!isWindows && hostProjectsFolder != null) {
return hostProjectsFolder;
if (!isWindows && singleMountPointForAllWorkspaces != null) {
return singleMountPointForAllWorkspaces;
}
return doGetPathById(workspaceId);
}

@Deprecated
String getPathByName(String workspaceName, String workspaceNamespace) throws IOException {
if (!isWindows && singleMountPointForAllWorkspaces != null) {
return singleMountPointForAllWorkspaces;
}

try {
Workspace workspace = workspaceDao.get(workspaceId);
String wsName = workspace.getConfig().getName();
return doGetPathByName(wsName);
Workspace workspace = workspaceDao.get(workspaceName, workspaceNamespace);
return getPath(workspace.getId());
} catch (NotFoundException | ServerException e) {
throw new IOException(e.getLocalizedMessage());
}
}

public String getPathByName(String workspaceName) throws IOException {
if (!isWindows && hostProjectsFolder != null) {
return hostProjectsFolder;
}
return doGetPathByName(workspaceName);
}
private String doGetPathById(String workspaceId) throws IOException {

private String doGetPathByName(String workspaceName) throws IOException {
final String workspaceFolderPath =
Paths.get(workspacesMountPoint).resolve(workspaceName).toString();
ensureExist(workspaceFolderPath, null);
String workspaceFolderPath = Paths.get(workspacesMountPoint).resolve(workspaceId).toString();
// Since Che may be running inside of container and workspaces are mounted not in the same way
// they will be mounted from host to workspace's containers we use Che master specific paths.
// In cases when Che master specific path is not defined it will be equal to the workspace's
// containers one.
String workspaceFolderPathForCheMaster =
Paths.get(workspacesMountPointForCheMaster).resolve(workspaceId).toString();
ensureExist(workspaceFolderPathForCheMaster, null);
return workspaceFolderPath;
}

@VisibleForTesting
@PostConstruct
void init() throws IOException {
// check folders creation flag from environment variable
String allowFoldersCreationEnvVar = System.getenv(ALLOW_FOLDERS_CREATION_ENV_VARIABLE);
if ("false".equalsIgnoreCase(allowFoldersCreationEnvVar)) {
createFolders = false;
}

// Priority of workspace storage path sources:
// If Che is running on Windows
// che-home-location/vfs
// Otherwise
// If environment variable for storage location is set
// use value of that variable
// Otherwise
// If old property of workspace storage is set
// use value of that property for backward compatibility
// Otherwise
// use up-to-date property
// find root directory for projects in workspaces
if (isWindows) {
final Path vfs = WindowsHostUtils.getCheHome().resolve("vfs");
workspacesMountPoint = vfs.toString();
// create directories if needed
if (singleMountPointForAllWorkspaces != null) {
ensureExist(singleMountPointForAllWorkspaces, SINGLE_MOUNTPOINT_FOR_ALL_WORKSPACES_PROPERTY);
} else {
String workspaceStorageFromEnv = System.getenv(WORKSPACE_STORAGE_PATH_ENV_VARIABLE);
if (workspaceStorageFromEnv != null) {
workspacesMountPoint = workspaceStorageFromEnv;
} else if (oldWorkspacesMountPoint != null) {
workspacesMountPoint = oldWorkspacesMountPoint;
}
}
ensureExist(workspacesMountPoint, WORKSPACE_PROJECTS_MOUNT_POINT_PROPERTY);

// create directories if needed
if (hostProjectsFolder == null) {
ensureExist(
workspacesMountPoint,
oldWorkspacesMountPoint == null
? "che.workspace.storage"
: "che.user.workspaces.storage");
} else {
ensureExist(hostProjectsFolder, "host.projects.root");
workspacesMountPointForCheMaster,
CHE_MASTER_SPECIFIC_WORKSPACE_PROJECTS_MOUNT_POINT_PROPERTY);

if (migrationOnStartup && !isWindows) {
localProjectsMigrator.performMigration(workspacesMountPointForCheMaster);
}
}
}

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,96 @@
/*
* Copyright (c) 2012-2017 Red Hat, Inc.
* All rights reserved. This program and the accompanying materials
* are made available under the terms of the Eclipse Public License v1.0
* which accompanies this distribution, and is available at
* http://www.eclipse.org/legal/epl-v10.html
*
* Contributors:
* Red Hat, Inc. - initial API and implementation
*/
package org.eclipse.che.workspace.infrastructure.docker.local.projects;

import static org.eclipse.che.api.core.Pages.iterate;

import com.google.inject.Inject;
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.nio.file.StandardCopyOption;
import java.util.HashMap;
import java.util.Map;
import java.util.Map.Entry;
import org.eclipse.che.api.core.ServerException;
import org.eclipse.che.api.workspace.server.model.impl.WorkspaceImpl;
import org.eclipse.che.api.workspace.server.spi.WorkspaceDao;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

/**
* Performs migration of workspace project files, that are stored in old format (in directories
* named after their workspace name), so they will be stored in folders named after their workspace
* ID.
*
* @author Mykhailo Kuznietsov
*/
public class LocalProjectsMigrator {

private final WorkspaceDao workspaceDao;

@Inject
public LocalProjectsMigrator(WorkspaceDao workspaceDao) {
this.workspaceDao = workspaceDao;
}

private static final Logger LOG = LoggerFactory.getLogger(LocalProjectsMigrator.class);

public void performMigration(String workspaceProjectsRootFolder) {
LOG.debug("Starting migration of workspace project files");
Copy link
Member

@sleshchenko sleshchenko Dec 22, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let's change it to info

Map<String, String> workspaceName2id = getId2NameWorkspaceMapping();
for (Entry<String, String> entry : workspaceName2id.entrySet()) {
Path workspaceStoredByNameLocation =
Paths.get(workspaceProjectsRootFolder).resolve(entry.getValue());
if (!Files.exists(workspaceStoredByNameLocation)) {
// migration is not needed for this workspace
continue;
}
LOG.debug(
"Performing migration of workspace with id '{}' and name '{}'",
entry.getKey(),
entry.getValue());
Path workspaceStoredByIdLocation =
Paths.get(workspaceProjectsRootFolder).resolve(entry.getKey());
try {
Files.move(
workspaceStoredByNameLocation,
workspaceStoredByIdLocation,
StandardCopyOption.ATOMIC_MOVE);
LOG.info(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd change info level here to debug.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no, stay with info. We need to provide this information to the user.

"Successfully migrated projects of workspace with id '{}' and name '{}'",
entry.getKey(),
entry.getValue());
} catch (IOException e) {
LOG.debug(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LOG.error

"Failed to migrate projects of workspace with id '{}' and name '{}'",
entry.getKey(),
entry.getValue());
}
}
}

private Map<String, String> getId2NameWorkspaceMapping() {
try {
Map<String, String> result = new HashMap<>();

for (WorkspaceImpl workspace :
iterate(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Iterate will eager fetch all objects from a database. I'd be nice to check how many memory it will consume when a number of workspaces is like 50 000 or the same as we have on codenvy.io.
If quite a lot then we should fetch workspace ids and names only. Something like that we've already done for another migration.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that's unnecessary, I don't think there will ever happen a migration of significant size of workspaces, ever.

(maxItems, skipCount) -> workspaceDao.getWorkspaces(false, maxItems, skipCount))) {
result.put(workspace.getId(), workspace.getConfig().getName());
}
return result;
} catch (ServerException e) {
throw new RuntimeException(e);
}
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -81,11 +81,11 @@ public void subscribe(EventService eventService) {
@Override
public void onEvent(WorkspaceRemovedEvent event) {
Workspace workspace = event.getWorkspace();
String workspaceName = workspace.getConfig().getName();
String workspaceId = workspace.getId();

String projectSourcesPath;
try {
projectSourcesPath = workspaceFolderPathProvider.getPathByName(workspaceName);
projectSourcesPath = workspaceFolderPathProvider.getPath(workspaceId);
} catch (IOException e) {
LOG.error(
"Failed to evaluate projects files path for workspace with id: '{}'. Cause: '{}'",
Expand Down
Loading